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(57) Abstract 

A control device, for example a computer mouse, has image-recording means which arc adapted to be moved, preferably manually, 
for conu-olling an object, for example a cursor on a computer screen, as a function of the movement of the image^rccording means. The 
control device is adapted to record a plurality of images widi partially overlapping contents when the image-recording means are being 
moved, the partially overlapping contents of the images enabling image-processing means to generate control signals indicating how the 
image-recording means have been moved. A method of controlling and a control device based on the turning of the image-recording means 
are also shown. 



FOR THE PURPOSES OF INFORMATION ONLY 



Codes used to identify States party to the PCT on the front pages of pamphlets publishing international applications under the PCT. 



AL 


Albania 


ES 


Spain 


LS 


Lesotho 


SI 


Slovenia 


AM 


Armenia 


FI 


Finland 


LT 


Lithuania 


SK 


Slovakia 


AT 


Austria 


FR 


France 


LU 


Luxembourg 


SN 


Senegal 


AU 


Australia 


OA 


Gabon 


LV 


Latvia 


SZ 


Swaziland 


AZ 


Azerbaijan 


GB 


United Kingdom 


MC 


Monaco 


TO 


Chad 


BA 


Bosnia and Herzegovina 


GE 


Georgia 


MD 


Republic of Moldova 


TG 


Togo 


BB 


Barbados 


OH 


Ghana 


MG 


Madagascar 


TJ 


Tajikistan 


BE 


Belgium 


ON 


Guinea 


MK 


The former Yugoslav 


TM 


Turkmenistan 


BF 


Builcina Faso 


OR 


Greece 




Republic of Macedonia 


TR 


Turkey 


BG 


Bulgaria 


HU 


Hungary 


ML 


Mali 


TT 


Tnnidad and Tobago 


BJ 


Benin 


IE 


Ireland 


MN 


Mongolia 


UA 


Ukraine 


BR 


Brazil 


IL 


Israel 


MR 


Mauritania 


UG 


Uganda 


BV 


Belarus 


IS 


Iceland 


MW 


Malawi 


US 


United States of America 


CA 


Canada 


IT 


Italy 


MX 


Mexico 


UZ 


Uzbekistan 


CF 


Central African Republic 


JP 


Japan 


NE 


Niger 


VN 


Viet Nam 


CG 


Congo 


KE 


Kenya 


NL 


Netherlands 


YU 


Yugoslavia 


CH 


Switzerland 


KG 


Kyrgyzstan 


NO 


Norway 


zw 


Zimbabwe 


CI 


Cflce d'lvoiie 


KP 


Democratic People's 


NZ 


New Zealand 






CM 


Cameroon 




Republic of Korea 


PL 


Poland 






CN 


China 


KR 


Republic of Korea 


PT 


Portugal 






CU 


Cuba 


KZ 


Kazakstan 


RO 


Romania 






CZ 


Czech Republic 


LC 


Saint Lucia 


RU 


Russian Federation 






D£ 


Germany 


LI 


Liechtenstein 


SD 


Sudan 






DK 


Denmark 


LK 


Sri Lanka 


SE 


Sweden 






E£ 


Estonia 


LR 


Liberia 


SG 


Singapore 







wo 99/60469 PCT/SE99/00719 



rONTROL DEVICE AND METHOD OF CONTROLLING AN OBJECT 

Field of the Invention 

The present invention relates to a control device 
having image -recording means which are adapted to be mov- 
ed, preferably manually, for controlling an object as a 
function of the movement of the image -recording means. 
5 The invention also relates to a method of controlling an 
object . 

Background of the Invention 

Today, personal computers are usually equipped with 
a control device, a so-called computer mouse, which is 

10 used for positioning a cursor on the computer screen. The 
positioning is carried out by the user passing the mouse 
over a surface, the hand movement indicating how the 
mouse should be positioned. The mouse generates position- 
ing signals indicating how the mouse has been moved and 

15 thus how the cursor should be moved. Presently, the most 
common type of mouse has a ball on its underside, which 
turns as a result of friction against the surface when 
the mouse is passed over the same and which in this con- 
nection drives position sensors which in turn generate 

20 the positioning signals. Normally, the mouse can also be 
used for providing instructions to the computer by the 
intermediary of one or more buttons on which the user 
clicks . 

Optical computer mouses are also known. JP 09190277 
25 shows an optical mouse having one CCD line sensor for 
the X-axis and one CCD line sensor for the Y-axis. Data 
recorded by means of the CCD line sensors at a certain 
time is compared with data recorded at a subsequent time, 
whereby the movement of the mouse in the X direction and 
30 in the Y direction can be determined. 

A mouse is thus used for controlling a virtual 
object. However, there are other control devices whose 
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structure is similar to that of a mouse, but which are 
used for controlling physical objects instead. 

Furthermore, there are control devices for control- 
ling objects in two dimensions, i.e. in a plane, or in 
5 three dimensions, i.e. in a space. 

WO 98/11528 describes a control device which pro- 
vides a computer with three-dimensional information. The 
device is based on three accelerometers which are placed 
in mutually perpendicular directions and which are cap- 

10 able of measuring acceleration or inclination in one to 
three directions. The device can, for example, be placed 
on the user's head or it can be hand-held. 

A computer mouse for inputting three-dimensional 
information to a computer is described in US Patent 

15 5,506,605. This computer mouse is hand-held and is 

intended to be held freely in space. Furthermore, it can 
comprise sensors for measuring various physical proper- 
ties which are subsequently interpreted by suitable elec- 
tronic means, converted into digital format, and input 

20 to the computer. The position of the mouse in space is 
determined by position sensors, which may be based on 
light, acceleration, gyroscopes, etc. In the embodiment 
described, use is made of an ultrasonic sensor and a 
magnetic sensor. On the basis of the input, the computer 

25 can subsequently generate tactile feedback in the form 
of vibrations which, for example, provide the user with 
information concerning the location of the mouse in rela- 
tion to its desired location. 
Summajry of tt^e Invention 

30 It is an object of the present invention to provide 

an improved control device and an improved method of con- 
trolling an object which are suited for both two-dimen- 
sional and three-dimensional control of physical as well 
as virtual objects. 

35 This object is achieved by control devices according 

to claims 1 and 23 and by a method according to claim 24. 
Preferred embodiments are stated in the subclaims . 
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Thus, according to a first aspect, the invention 
relates to a control device having image -recording means 
which are adapted to be moved by a user, preferably 
manually, for controlling an object, which may be physi- 
5 cal or virtual, as a function of the movement of the 
image -recording means. The image -recording means are 
adapted to record a plurality of images with partially 
overlapping contents when they are being moved, the 
partially overlapping contents enabling the determination 

10 of how the image -recording means have been moved. 

The invention is thus based on the idea of using 
images for determining how a unit is being moved. This 
technology can be used for two-dimensional as well as 
three-dimensional control. It is advantageous because 

15 it requires few sensors and no moving parts. The entire 
movement information is contained in the overlapping con- 
tents of the images. Because the device records images of 
the surroundings, an "absolute" position indication is 
obtained, making it possible to detect when the image- 

20 recording means are in a specific position, which, for 

example, is not possible when using control devices based 
on measuring acceleration. In addition to movement, turn- 
ing can also be detected and used for controlling an 
object . 

25 In one embodiment, the control device is designed 

for controlling an object in a plane. In this case, the 
overlapping images enable the determination of not only 
the movement of the image -recording means but also their 
turning in the plane, which, for example, is not possible 

30 when using a traditional mouse with a ball. Accordingly, 
the control device is advantageously adapted to control 
the angular position of the object in the plane. When the 
device is designed for control in a plane, the image- 
recording means are advantageously provided with a light - 

35 sensitive sensor means having a two-dimensional sensor 
surface, a so-called area sensor, for recording the 
images. In this context, a two-dimensional sensor sur- 
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face refers to the fact that the sensor surface must be 
capable of imaging a surface with a matrix of pixels. CCD 
sensors and CMOS sensors are examples of suitable sen- 
sors. A single sensor is thus sufficient for providing 
5 control in a plane. 

In an alternative embodiment, the device is designed 
for controlling an object in a space. In this case, too, 
the control device is advantageously adapted to control 
the angular position of the object, in which connection 

10 the control can take place about three axes. In an econo- 
mical embodiment, it may be sufficient for the device to 
have two light-sensitive sensors each having a two-dimen- 
sional sensor surface for recording said images in two 
different directions. 

15 However, for more precise control in space, it is 

preferable for the image -recording means to comprise 
three sensors for recording the images in three, 
preferably perpendicular, directions. This enables the 
determination of the translation along three mutually 

20 perpendicular axes as well as of the rotation about these 
axes by means of relatively simple calculations. 

Suitably, the control device has image -processing 
means for providing control signals for controlling the 
object. The image-processing means may be located in the 

25 same physical casing as the image -recording means, the 
output signals from this physical casing thus constitut- 
ing the control signals for controlling the object which 
is to be controlled. However, the image -processing means 
may also be located in another physical casing, for exam- 

30 pie in a computer whose cursor constitutes the object 

which is to be controlled, or in a computer which in turn 
controls, or forms part of, a physical object which is 
controlled by means of the control device, the output 
signals from the image-processing means constituting the 

35 control signals for controlling the object. In this con- 
text, it should be noted that the control signals output - 
ted from the image -processing means may require further 
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processing before they can be used for direct control of 
the object. The image -processing means are advantageously 
implemented with the aid of a processor and software, but 
can also be implemented completely with the aid of hard- 
5 ware • 

The image-processing means are suitably adapted to 
determine the relative positions of the images with the 
aid of the partially overlapping contents for providing 
said control signals. If the control device is used for 

10 control in three dimensions, this is suitably carried out 
in parallel with respect to all the sensors. The distance 
and direction of the movement, and thus the current 
position, can be determined on the basis of the relative 
positions of the images. 

15 Advantageously, the control device has a calibration 

mode, in which the image -recording means are moved in a 
way that enables the image-processing means to relate the 
relative positions of the images to an actual movement of 
the image -recording means. As an alternative, the control 

20 device could be provided with a distance meter measuring 
the distance to the surfaces being imaged with the aid 
of the sensors, but that would, of course, be more expen- 
sive . 

The image -processing means are suitably adapted to 
25 generate said control signals on the basis of at least 
one movement vector obtained from the relative positions 
of the images. 

Additionally, or alternatively, the image -processing 
means may be adapted to generate said control signals on 
30 the basis of at least one turning indication obtained 
from the relative position of the images. The control 
signals can thus be used for controlling the turning of 
an object as well as its movement, which is an advantage 
compared to traditional mechanical computer mouses. 
35 In the case of a control device for three-dimen- 

sional control, the image-processing means can combine 
information from all the sensors with respect to the 
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relative positions of the images in order to generate one 
movement vector and one turning vector. In this way, the 
position of the image-recording means can be unambiguous- 
ly determined. In other words, the control device can 
5 carry out a digitisation of the movement performed by a 
hand when it moves the image -recording means in order to 
enable a computer to control an object on the basis of 
this movement. 

In one embodiment, the image -processing means may be 

10 adapted to generate said control signals on the basis of 
the speed at which the image -recording means have been 
moved, the speed being determined from the relative posi- 
tions of the images and the image -recording frequency. 

Suitably, the receiver of the control signals should 

15 know that the control signals are control signals so that 
it will know how the signals are to be subsequently pro- 
cessed. Consequently, the image -processing means are pre- 
ferably adapted to output said control signals in such a 
way that a receiver can identify the control signals as 

20 being intended for controlling an object. This can, for 
example, be effected by the use of a predetermined proto- 
col . 

An advantage of using an image-based control device 
is that it becomes possible to determine when the image- 

25 recording means are in a predetermined position, since 
this position can be defined by means of one or several 
images. For example, it is possible to detect when the 
image -recording means have returned to their original 
position. For this purpose, the image -processing means 

30 are adapted to store at least one reference image and to 
compare images recorded subsequently with this image in 
order to generate a signal in the case of an essentially 
complete overlap. For instance, the user can define a 
certain position as a reference position by clicking on 

35 the control device in this position. 

If the image -recording means and the image -process- 
ing means are located in different physical casings. 
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the image -recording means may advantageously comprise 
a transmitter for wireless transmission of images from 
the image -recording means to the image-processing means. 
Moreover, especially if the image -recording and the 
5 image-processing means are located in the same physical 
casing, it may be an advantage if the image -processing 
means comprise a transmitter for wireless outputting of 
the control signals, for example to a computer whose 
cursor is to be controlled. In both cases, the control 

10 device is very easy to use since no flex is required for 
the information transfer. For example, a user can have a 
personal image -recording means or control device and use 
it with different computers or receivers of the control 
signals. The transmitter can be an IR transmitter, a 

15 radio transmitter, which, for example, uses the so-called 
Bluetooth standard, or some other transmitter which is 
suitable for wireless information transfer between two 
units located fairly close to each other. 

In a preferred embodiment, the control device is a 

20 computer mouse, i.e. a device which can be connected to 
a computer and be used for positioning a cursor in one, 
two, or several dimensions. 

The control device can be used in a first absolute 
mode or in a second relative mode. In the absolute mode, 

25 the movement of the controlled object is proportional 
to the movement of the image -recording means. In other 
words, the object moves in a way that corresponds to the 
movement of the image -recording means, regardless of 
where these are located. In the relative mode, however, 

30 the control device is configured so that the speed or 

acceleration of the controlled object increases when the 
distance increases between the image -recording means and 
a predefined origin of coordinates. In this way, it 
becomes possible to achieve faster movement of the object 

35 by holding the image -recording means farther away from 

the predefined origin, while, at the same time, precision 
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control can be achieved by holding the image -recording 
means closer to the origin. 

According to a second aspect of the invention, it 
relates to a control device having image -recording means 
5 which are adapted to be turned, preferably manually, for 
controlling an object as a function of the turning of the 
image -recording means. The control device is adapted to 
record a plurality of images with partially overlapping 
contents when the image -recording means are being turned, 
10 the partially overlapping contents of the images enabling 
the determination of how the image -recording means have 
been turned. 

This control device is thus based on the same idea 
as the control device described above, but instead of 

15 controlling the object as a function of the movement of 
the image -recording means, it is controlled as a function 
of their turning. This control device may, for example, 
be a trackball. The embodiments discussed above are to 
a large extent also applicable to the turning control 

20 device, and the same advantages are obtained. 

According to a third aspect of the invention, it 
relates to a method of controlling an object, comprising 
the steps of moving a control device; recording, with the 
aid of the control device, a plurality of images with 

25 overlapping contents during the movement of the control 
device; and determining the movement of the control 
device with the aid of the contents of the overlapping 
images. The same advantages are obtained as those 
described with respect to the above-mentioned devices. 

30 Brief Description of the Drawings 

The present invention will be described in more 
detail below by way of exemplifying embodiments with 
reference to the accompanying drawings, in which 

Fig. 1 schematically shows an embodiment of a con- 

35 trol device according to the invention; 
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Fig. 2 is a block diagram of the electronic cir- 
cuitry part of an embodiment of the control device 
according to the invention; 

Fig. 3 schematically shows a second embodiment of 
5 a control device according to the inventions- 
Fig. 4 is a flowchart illustrating the operation of 
a control device for two-dimensional control; 

Fig. 5 schematically shows an "open box" in which 
the control device in Fig. 3 can be used; 
10 Fig. 6 schematically shows a movement of the control 

device according to the invention from a point (x,y,z) to 
a point (x+5x, y+6y, z+8z) in an orthonormal coordinate sys- 
tem with the axes ex, ey, and ez; 

Fig. 7 schematically shows which translation scalars 
15 are outputted from the respective sensors when the con- 
trol device is being moved (index shows which sensor is 
generating the respective scalars) ; and 

Fig. 8 schematically shows how the control device is 
intended to be moved in the calibration mode. 
20 Description of Preferred Embodiments 

The control device according to the invention can be 
implemented in embodiments of essentially two main types. 
A first embodiment of the control device according to the 
invention will be described below, which embodiment is 
25 intended to be used as a two-dimensional mouse. Next, a 
second embodiment of the control device will be describ- 
ed, which embodiment is intended to be used as a three- 
dimensional mouse. Finally, the operation of the two- 
dimensional and the three-dimensional mouse will be 
30 described. In both embodiments described, the image- 
recording means and the image-processing means are locat- 
ed in the same physical casing, from which control sig- 
nals are outputted. As mentioned above, the image-pro- 
cessing means can also be located in a separate physical 
35 casing. It is very simple for the skilled person to carry 
out this modification. 
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Design of the Control Device 

In the first embodiment of the control device shown 
in Fig. 1, it comprises a casing 1 having approximately 
the same shape as a conventional highlighter pen. One 
5 short side of the casing has a window 2, by the inter- 
mediary of which images are read into the device. The 
window 2 is somewhat recessed in the casing in order not 
to wear against the underlying surface. 

The casing 1 essentially contains an optics part 3, 
10 an electronic circuitry part 4, and a power supply part 
5. 

The optics part 3 comprises a light-emitting diode 
6, a lens system 7, and image -recording means in the form 
of a light-sensitive sensor 8, which constitutes the 
15 interface with the electronic circuitry part 4 . 

The task of the LED 6 is to illuminate a surface 
which is currently located under the window in the case 
where the control device is held directly against a sur- 
face or very close thereto. A diffuser 9 is mounted in 
20 front of the LED 6 for diffusing the light. 

The lens system 7 has the task of projecting an 
image of the surface located under the window 2 onto 
the light-sensitive sensor 8 as accurately as possible. 

In this example, the light-sensitive sensor 8 com- 
25 prises a two-dimensional, square CCD unit (CCD = charge 
coupled device) with a built-in A/D converter. Such sen- 
sors are commercially available. The sensor 8 is mounted 
at a small angle to the window 2 and on its own circuit 
board 11. 

30 The power supply to the control device is obtained 

from a battery 12, which is mounted in a separate 
compartment 13 in the casing. 

The block diagram in Fig. 2 schematically shows the 
electronic circuitry part 4. This is located on a circuit 

35 board and comprises a processor 20, which by the interme- 
diary of a bus 21 is connected to a ROM 22, in which the 
programs of the processor are stored, to a read/ write 
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memory 23, which constitutes the working memory of the 
processor and in which the images from the sensor are 
stored, to a control logic unit 24, as well as to the 
sensor a: and the LED 6. The processor 20, the bus 21, 
5 the memories 22 and 23, the control logic unit 24, as 
well as the associated software together constitute 
image-processing means. 

The control logic unit 24 is in turn connected to 
a number of peripheral units, comprising a radio trans - 

10 ceiver 26 for transferring information to/from an exter- 
nal computer, buttons 27, by means of which the user can 
control the image -recording means and which can also be 
used as the clicking buttons of a traditional mouse, as 
well as an indicator 29, e.g. a LED, indicating when the 

15 mouse is ready to be used. Control signals to the memo- 
ries, the sensor, and the peripheral units are generated 
in the control logic unit 24, The control logic also 
handles generation and prioritisation of interrupts to 
the" processor . The buttons 27, the radio transceiver 26, 

20 and the LED 6 are accessed by the processor writing and 
reading in a register in the control logic unit 24. The 
buttons 27 generate interrupts to the processor 20 when 
they are activated. 

Fig. 3 shows a second embodiment of the control 

25 device according to the invention. Like the first embodi- 
ment, this embodiment comprises a pen-shaped casing 31. 
Besides the window 32 on one short side of the casing, 
the device has two additional windows 32' and 32". Each 
of the windows 32, 32 32" is somewhat recessed in the 

30 casing so that it will not wear or scratch should the 

control device impinge upon a surface when it is in use, 
or when it is in the idle position. 

As in the above case, the casing 1 essentially con- 
tains an optics part 33, an electronic circuitry part 34, 

35 and a power supply part 5. 

The optics part 33 comprises a lens package (not 
shown) with three lens systems and a set of sensors (not 
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shown) with three light-sensitive sensors which consti- 
tute the interface to the electronic circuitry part 34 
for the windows 32, 32' and 32" respectively. There is 
no light -emitting diode in this embodiment. The control 
5 device is intended to be held at a distance from the sur- 
faces which are being imaged and, consequently, in most 
cases, ambient light is sufficient to permit images to be 
recorded . 

The lens systems have the task of projecting images 
10 of the surfaces at which the windows 32, 32', 32" are 
directed onto the light-sensitive sensors as accurately 
as possible. 

As in the above embodiment, the light-sensitive sen- 
sors comprise two-dimensional, square CCD units with 

15 built-in A/D converters. Each sensor is mounted on its 
own circuit board. 

In this embodiment, too, the power supply to the 
control device is obtained from a battery, which is 
mounted in a separate compartment in the casing. 

20 In this second embodiment, the design of the elec- 

tronic circuitry part is essentially the same as that 
described above with respect to the first embodiment. The 
electronic circuitry part is shared by all three sensors. 
Application of the Device as a Two-dimensional Mouse 

25 The device according to the first embodiment can be 

used as a mouse for inputting movement information, by 
means of which a cursor can be controlled on a computer 
screen. 

The user directs the window 2 of the control device 
30 at a patterned surface, e.g. a mouse pad. He presses one 
of the buttons 27 to activate the image -recording means, 
whereupon the processor 20 commands the LED 6 to begin 
generating strobe pulses at a predetermined frequency, 
suitably at least 50 Hz. Subsequently, the user passes 
35 the control device over the surface in the same way as 
if it were a traditional mouse, whereupon images with 
partially overlapping contents are recorded by the sensor 
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8 and stored in the read/write memory 23. The images are 
stored as images, i.e. with the aid of a plurality of 
pixels, each having a grey scale value in a range from 
white to black. 
5 The flowchart in Fig. 4 shows the operation of the 

two-dimensional mouse in more detail. In step 400, a 
starting image is recorded. In step 401, the next image 
is recorded. The contents of this image partially overlap 
the contents of the previous image. 

10 As soon as an image has been recorded in step 401, 

the process begins of determining how it overlaps the 
previous image, step 402, i.e. in which relative position 
the best match is obtained between the contents of the 
images. This determination is carried out by translating 

15. the images vertically and horizontally relative to each 
other, and by rotating the images relative to each other. 
For this purpose, every possible overlap position between 
the images is examined, at the pixel level, and an over- 
lap measurement is determined as follows: 

20 1) For each overlapping pixel position, the grey 

scale values of the two relevant pixels are added up if 
the latter are not white. Such a pixel position in which 
none of the pixels is white is designated a plus posi- 
tion. 

25 2) The grey scale sums for all the plus positions 

are added up. 

3) The neighbours of each pixel position are exa- 
mined. If an overlapping pixel position is not a neigh- 
bour of a plus position and consists of a pixel which is 

30 white and a pixel position which is not white, the grey 
scale value of the non- white pixel is subtracted, pos- 
sibly multiplied by a constant, from the sum in point 2) . 

4) The overlap position providing the highest over- 
lap measurement as stated above is selected. 

35 Our Swedish patent application No. 9704 924-1 and the 

corresponding U.S. application No. 024 641 describe an 
alternative way of matching the images in order to find 
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the best overlap position. The content of these applica- 
tions is herewith incorporated by reference. 

As soon as the best overlap position between the 
current image and the previous image has been determined, 
5 the previous image is discarded, whereupon the current 
image becomes the previous image in relation to the next 
image recorded. 

By determining the relative position of the two 
images a movement vector is obtained, which indicates how 

10 far and in which direction the image -recording means have 
been moved between the recording of the two images. If 
the mouse has also been turned between the two images, 
also a measurement of this turning is obtained. Subse- 
quently, a control signal, which includes the movement 

15 vector and the measurement of turning, is transmitted, 

step 403, by the radio transceiver 26 to the computer for 
which the control device is operating as a mouse. The 
computer uses the movement vector and the measurement of 
turning for positioning the cursor on its screen - 

20 Subsequently, the flow returns to step 401. In order to 
increase the speed, the steps can be partly carried out 
in parallel, e.g. by starting the recording of the next 
image while the current image is being put together with 
the previous image . 

25 When the mouse is activated, the buttons 27 can be 

used as clicking buttons for inputting instructions to 
the computer. 

Application o f the Device as a Three-dimensional Mouse 

The device according to the second embodiment can be 

30 used as a mouse for inputting movement information, by 

means of which a cursor can be controlled in three dimen- 
sions on a computer screen, i.e. in a space. 

As described above, the three-dimensional mouse com- 
prises three sensors 32, 32', 32" having two-dimensional, 

35 light-sensitive sensor surfaces. The main axes of the 

sensors are orientated along the x- , y- , and z-axes in an 
orthogonal coordinate system and have a two-dimensional 
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spatial resolution of n x n pixels and a time resolution 
of m images per second. Each lens system provides a field 
of vision with an angle of vision of v radians for the 
associated sensor surface. 
5 When the device is in use, the mouse movements are 

carried out in an "open box" 50 according to Fig, 5, 
which is defined by at least two side walls 51 and 52 
which are orientated at right angles in relation to each 
other, and a floor 53, It is also possible that the mouse 

10 can be held freely in space, but this requires more com- 
plicated calculation algorithms than the ones that will 
be described below. 

When the device is in use, the above method of 
determining the relative position of images is used for 

15 each sensor. Accordingly, the operation in this case can 
also be described by means of the flowchart in Fig. 4, 
but instead of recording individual images, a set of 
images consisting of three images is recorded simulta- 
neously. One movement vector and one turning vector are 

20 thus generated with the aid of the images recorded by 
each light-sensitive sensor, which vectors describe the 
movement carried out by the mouse between the recording 
of two consecutive images. These vectors are then includ- 
ed in a control signal which is transmitted to the object 

25 which is to be controlled by means of the mouse. 

Furthermore, to enable the successful use of the 
mouse, it is necessary that the light conditions be such 
that the light-sensitive sensors are capable of recording 
images of sufficiently high quality to permit their pro- 

30 cessing as described above. 

In order further to facilitate the reader's under- 
standing of how the movement of the mouse can control the 
object, a description of the calculations carried out to 
determine the movement of the mouse will now be provided 

35 by way of example, with reference to Fig. 6. In the cal- 
culations below it is assumed that the image -matching 
algorithm is of a simple type which for each sensor mere- 
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ly calculates the translation in two mutually perpendicu- 
lar directions between two images. Suppose that the mouse 
is located in the position (x, y, z) and that it has a 
rotation which can be described by means of the ortho- 
5 normal rotation matrix R, The x-axis of the mouse thus 
points in the direction Re^, the y-axis points in the 
direction R-ey, and the z-axis points in the direction 
R-ez. Also suppose that between the recording of two 
images, the mouse carries out a translational motion 
10 and/or a rotational motion according to: 

(x,y,z) -> (x+5x,y+5y, z+8z) 
R R • 5R 

In the local coordinate system of the mouse, the transla- 
tion vectors can be defined as shown in Fig. 7. The first 

15 sensor records movements in the x and y directions, the 
second sensor records movements in the y and z direc- 
tions, and the third sensor records movements in the x 
and z directions. Consequently, for any triplet of conse- 
cutive images, the translation scalars (xl, yl, y2, z2, 

20 x3, z3) describe the detected movement of the mouse. The 
translation scalars consist of the outputs from the 
image -matching algorithm for each sensor. 

In order to calculate the rotation of the mouse, the 
effect of a rotation upon the translation scalars is cal- 

25 culated. Suppose that the mouse is rotated through an 
angle a, which is sufficiently small so that sin a = a. 
For the sake of clarity, it is also assumed that the 
rotation takes place about the z-axis by az radians. This 
rotation results in the scalars 

30 

v2 = — a. and jc3 = — a, 

V V 

where n is the number of pixels along one side of the 
sensor and v is the angle of vision of the sensor sur- 
face, expressed in radians. Thus, the following applies 
35 to all the axes: 
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By knowing the values of the translation scalars, 
which are output signals from the image -matching algo- 
5 rithm, the number of pixels n of the sensor surface 

along one side and the angle of vision v of the sensor, 
it is thus possible to calculate a rotation vector (ax, 
ay, az) for the rotation of the mouse about the x-, y- 
and z-axes. 

10 Furthermore, in order to calculate the translational 

motion one must know the functional distance from each 
sensor to the ambient geometry. The functional distance 
is a constant which relates the output from the. image- 
matching algorithm to the translational motion. The 

15 functional distance is determined by misans of a calibra- 
tion which will be described below. In the special case 
where the mouse moves inside an "open box" 50 as describ- 
ed above, the functional distance corresponds to the geo- 
metrical distance from the middle of the mouse to the 

20 respective walls 51, 52, and 53 of the box 50. 

For the sake of clarity, a translation of a distance 
8x along the x-axis is looked at. Accordingly, the effect 
of the translation with respect to the scalars xl and x3 
will be respectively 

25 

x\ = Sx 

2d, tan- 
' 2 
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x3 = ' 



2d^ tan- 



Here, di and ds are the functional distances from the 
mouse to the projected surfaces with respect to (xl,yl) 
and (x3/Z3). The following is obtained if this is gene- 
ralised to all the axes 



n 



2d. tan— 
' 2 



2d. tan- 



2d, tan— 



n 



2rf, tan- 
' 2 



tan- 



2^3 tan- 



Sy 
Sz 



x\ 

z2 
x3 
z3 



10 



15 



20 



By knowing the values of the translation scalars, 
which are obtained as output signals from the image- 
matching algorithm, the number of pixels n of the sensor 
surface along one side, the field of vision v of the sen- 
sor and the functional distances d^- to the projected 

surfaces, it is thus possible to calculate a translation 
vector (5x,5y,5z) for the translation of the mouse along 
the X- , y- and z-axes. 

To sum up, the translation scalars (xl, yl, y2, z2, 
x3, z3) which are obtained in the image matching, thus 
depend on the rotation as well as translation of the 
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mouse. Knowing these and other parameters described 
above, a translation vector {5x,6y,5z) and a rotation 
vector (ax/tty^ttz) can be obtained by solving the following 
system of equations, which is solvable. These vectors are 
then included in a control signal which is transmitted to 
the object controlled by means of the mouse, which signal 
indicates the new position of the object. 



n 



2d. tan- 
' 2 



2d. tan— 



2d, tan- 



2d^ tan— 



n 



2rf, tan- 
' 2 
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Calibration, i.e. the calculation of the functional 
distances di, d2, and da, can be carried out by moving the 
mouse along the edges of the open box. The mouse is moved 
along the x-, y-, and z-axes according to a sequence A-B- 
C shown in Fig. 8. Each movement gives rise to two equa- 
tions, which together give the following system of equa- 
tions : 

n 



d,-0,5 = 



^3+0.5 = 



2jc, tan- 
* 2 
n 

V 

2x. tan— 
' 2 



^2-0,5 = . 



2y^ tan- 



2y^ tan— 



^2-0,5 = - 



^3-0,5 = - 



n 

y 

2zy tan— 
' 2 
n 

2z, tan— 
' 2 
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This overdefined system of equations contains all 
the information required to calculate the values of the 
functional distances di, d2, and da. 

Furthermore, by means of the mouse according to this 
5 embodiment, the user can choose at a certain time to 

store the images which the sensors are currently record- 
ing in a memory. Subsequently, each set of recorded 
images is compared to the stored set of images and when 
a complete overlap exists, a signal is generated to the 

10 user. This enables precision control of an object since 
the user can find his way back to the exact position at 
which the mouse was located on a previous occasion. 
Naturally, the same principles can be used in the case of 
two-dimensional control of an object. 

15 In another application of the mouse, only the rota- 

tional motion is detected. In this case, no calibration 
is required and it is sufficient to solve the equation 
presented above in connection with the discussion con- 
cerning rotation. In this application, the mouse can, for 

20 example, be mounted on a helmet or the like which is worn 
by the user and which, for example, is used in various 
types of virtual reality applications. 
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CLAIMS 

1. A control device having image -recording means 

5 which are adapted to be moved, preferably manually, for 
controlling an object as a function of the movement of 
the image -recording means; characterised in 
that the control device is adapted to record a plurali- 
ty of images with partially overlapping contents when 
10 the image -recording means are being moved, the partially 
overlapping contents of the images enabling the determi- 
nation of how the image -recording means have been moved. 

2. A control device according to claim 1, wherein 
the control device is adapted to control said object in 

15 a plane. 

3. A control device according to claim 2, wherein 
the control device is adapted to control the angular 
position of said object in said plane. 

4. A control device according to claim 2 or 3, 
20 further comprising a light-sensitive sensor means (8) 

having a two-dimensional sensor surface for recording 
the images . 

5. A control device according to claim 1, wherein 
the control device is designed for controlling said 

25 object in a space. 

6. A control device according to claim 5, wherein 
the control device is adapted to control the angular 
position of said object in said space. 

7. A control device according to claim 5 or 6, 

30 further comprising at least two light-sensitive sensor 
means (8) having a two-dimensional sensor surface for 
recording said images in two different directions. 

8. A control device according to claim 5 or 6, 
further comprising three light-sensitive sensor means 

35 (8) having a two-dimensional sensor surface for record- 
ing said images in three linearly independent directions. 
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9. A control device according to any one of claims 
1-8, further comprising image -processing means (20-24) 
for providing control signals for controlling said 
object. 

5 10. A control device according to claim 9, wherein 

the image-processing means (20-24) are adapted to deter- 
mine the relative positions of the images with the aid 
of the partially overlapping contents for providing said 
control signals. 

10 11. A control device according to any one of claims 

5-8, further comprising image -processing means (20-24) 
which are adapted to determine the relative positions of 
the images with the aid of the partially overlapping con- 
tents simultaneously with respect to all the light-sensi- 

15 tive sensor means (8) for providing said control signals, 

12 . A control device according to claim 11 , the con- 
trol device furthermore having a calibration mode, in 
which the image -recording means are moved in a way that 
enables the image -processing means (20-24) to relate the 

20 relative positions of the images to an actual movement of 
the image -recording means. 

13 . A control device according to any one of claims 
10-12, wherein the image-processing means (20-24) are 
adapted to generate said control signals on the basis of 

25 at least one movement vector obtained from the relative 
positions of the images. 

14 . A control device according to any one of claims 
10-13, wherein the image -processing means (20-24) are 
adapted to generate said control signals on the basis of 

30 at least one turning indication obtained from the rela- 
tive positions of the images. 

15. A control device according to any one of claims 
10-14, wherein the image-processing means (20-24) are 
adapted to generate said control signals on the basis of 

35 the speed at which the image -recording means have been 

moved, the speed being determined from the relative posi- 
tions of the images. 



wo 99/60469 PCT/SE99/00719 

23 

16. A control device according to claims 9-15, 
wherein the image-processing means (20-24) are adapted 
to output said control signals in a way that enables a 
receiver to identify the control signals as being intend- 

5 ed for controlling an object, 

17. A control device according to any one of claims 
9-16, wherein the image-processing means (20-24) are fur- 
thermore adapted to store at least one reference image 
and to compare images recorded subsequently with that 

10 image in order to generate a signal in the case of an 
essentially complete overlap. 

18. A control device according to any one of claims 
9-17, wherein the image -processing means (20-24) comprise 
a transmitter (26) for wireless outputting of the control 

15 signals. 

19. A control device according to any one of claims 
9-18, wherein the image -recording means comprise a trans- 
mitter (26) for wireless transmission of images to the 
image -processing means (20-24) . 

20 20. A control device according to any one of the 

preceding claims, wherein the control device is a mouse. 

21. A control device according to any one of the 
preceding claims, wherein the device has a first operat- 
ing mode in which the control device is adapted to con- 

25 trol said object in a way that enables its movement to 
be proportional to the movement of the image -recording 
means . 

22 . A control device according to any one of the 
preceding claims, wherein the device has a second operat- 

30 ing mode in which the control device is adapted to con- 
trol said object so that the speed of its movement is 
proportional to the distance between the image -recording 
means and a predefined origin. 

23. A control device having image -recording means 
35 which are adapted to be turned, preferably manually, for 

controlling an object as a function of the turning of the 
image -recording means; characterised in that 
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the control device is adapted to record a plurality of 
images with partially overlapping contents when the 
image -recording means are being turned, the partially 
overlapping contents of the images enabling the determi- 
5 nation of how the image -recording means have been turned. 

24. A method of controlling an object, comprising 
the steps of 

- moving a control device; 

- recording, with the aid of the control device, a 
10 plurality of images with overlapping contents during the 

movement of the control device, and 

- determining the movement of the control device 
with the aid of the contents of the overlapping images. 

25. A method of controlling an object according to 
15 claim 24, further comprising the step of 

- determining the relative position of the images 
with the aid of the partially overlapping contents for 
providing control signals for controlling the object. 
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